CS-466/566: Math for AI

Module 07: Deep Learning Fundamentals-3

Dr. Mahmoud Mahmoud

The University of Alabama

2026-03-26

TABLE OF CONTENTS

1. Neural Networks for Classification/Regression●

2. The Softmax Cross Entropy Loss Function○

3. Multi-class network: computational graph○

Neural Networks Definition (a.k.a. Deep Learning)

A deep learning model is a computational graph that try to map inputs, each drawn from some dataset with common characteristics to outputs drawn from a related distribution.

It is a graph that has many layers (Simply Nested Functions).

That is why gradient descent is a bit tricky here.

This slide defines deep learning as a computational graph with many layers.

Regression vs Classification Neural Networks

Classification

Use a sigmoid (or another squashing nonlinearity) on a single output so the readout behaves like a probability or score — that setup is classification.

Regression

Use a linear readout from the hidden layer, e.g. \(\sum_i w^2_{i,1}\, H_i\), to predict a real number — that setup is regression.

This slide compares regression and classification neural networks with their respective output functions.

Binary Classification

Binary Classification Neural Network

This slide shows a neural network diagram for binary classification.

(One vs All) Multi-class Classification

Given a classification problem with N possible solutions, a one-vs.-all solution consists of N separate binary classifiers

Note. The output vector has to be encoded using one hot encoding

This slide explains the one-vs-all approach for multi-class classification using multiple binary classifiers.

Training a Neural Network

• L is the loss function.
• Y is the target value.
• \(L(Y, O)\) is the loss between the target value and the output.
• We need to find each \(\frac{\partial L}{\partial w^k_{ij}}\)
• Calculus chain rule is needed to find the derivatives.

TABLE OF CONTENTS

1. Neural Networks for Classification/Regression✓

2. The Softmax Cross Entropy Loss Function●

3. Multi-class network: computational graph○

More Accurate Multi-class Classification

Previously, we used sigmoid with mean squared error (MSE) as loss function

• Pros
  • Convex (i.e., steeper gradient as you are away from the correct output)
  • Acceptable performance
• Cons
  • Outputs can not be interpreted as probability.
  • Does not work with all problem types

In practice, we use Softmax + cross Entropy loss

This slide compares the limitations of sigmoid with MSE and introduces the need for softmax with cross-entropy loss.

Softmax Function

• Softmax extends binary logistic regression idea (probability adds up to 1) into a multi-class world.
• It helps training converge more quickly than it otherwise would.

Let \(z\) denote the logits (the raw, unnormalized scores output by the network before applying softmax).

Softmax Function

This slide introduces the softmax function and its role in multi-class classification.

How it works?

Suppose we have an input vector from the previous layer

[5, 3, 2]

One way to transform these values into a vector of probabilities

A better way

This slide demonstrates different approaches to converting raw scores to probabilities, showing the superiority of softmax.

Intuition

Softmax Intuition

• Softmax is more strongly amplifies the maximum value relative to the other values.
• Softmax is partway between normalizing the values and actually applying the max function!

This slide provides intuition about how softmax amplifies the maximum value while maintaining probability interpretation.

Our Neural Network

Neural Network Architecture

This slide shows the complete neural network architecture with softmax and cross-entropy loss.

Loss Function Diagram View

• The incoming vector \(\textbf{x}\) is transformed by the softmax function to produce the vector of probabilities \(\textbf{p}\).
• The vector of probabilities \(\textbf{p}\) is then compared to the vector of actual values \(\textbf{y}\) using the cross entropy loss function resulting in a scalar loss value.

This slide shows the computational flow from input through softmax to cross-entropy loss calculation.

Cross Entropy

Recall that any loss function will take in a vector of probabilities and a vector of actual values

The cross entropy loss function, for each index i in these vectors, is:

\(\ell_i\) = \(-y_ilog(p_i)-(1-y_i)log(1-p_i)\)

This slide defines the cross-entropy loss function for individual elements.

Intuition

• Since \(p_i\) is a probability between 0 and 1.
• Our loss can be defined as follows.
  • if \(y_i = 1\) : Loss = \(-log(p_i)\)
  • if \(y_i = 0\) : Loss = \(-log(1 - p_i)\)

Cross Entropy Loss Intuition

This slide provides intuition about how cross-entropy loss penalizes incorrect predictions.

Gradient Computation

The real magic happens when we combine this loss with the softmax function

\(-\ell_1 = y_1 \log \left( p_1 \right) + (1-y_1) \log \left(1 - p_1 \right)\)

\(\Downarrow\)

\(-\ell_1 = y_1 \log \left(\frac{e^{x_1}}{e^{x_1}+e^{x_2}+e^{x_3}}\right) + (1-y_1) \log \left(1 - \frac{e^{x_1}}{e^{x_1}+e^{x_2}+e^{x_3}}\right)\)

Based on this expression, the gradient would seem to be a bit trickier for this loss. (Proof Is Required)

\(\displaystyle \frac{\partial L_i}{\partial x_i} = p_i - y_i\)

This slide shows the complex expression for cross-entropy loss with softmax and reveals the surprisingly simple gradient. We state the gradient for the minimized loss \(L=-\ell\) (standard in optimization).

Proof: \(\frac{\partial L}{\partial x} = p - y\)

\(-\ell_1 = y_1 \log \left(\frac{e^{x_1}}{e^{x_1}+e^{x_2}+e^{x_3}}\right) + (1-y_1) \log \left(1 - \frac{e^{x_1}}{e^{x_1}+e^{x_2}+e^{x_3}}\right)\)

Proof.

1. Split logarithm of fraction for first term
   

   \(-\ell_1 = y_1 \left[ \log(e^{x_1}) - \log(e^{x_1}+e^{x_2}+e^{x_3}) \right] + (1-y_1) \log \left[1 - \frac{e^{x_1}}{e^{x_1}+e^{x_2}+e^{x_3}}\right]\)

2. Simplify \(\log(e^{x_1}) = x_1\quad\quad\) and rewrite complement fraction
   

   \(-\ell_1 = y_1 x_1 - y_1 \log(e^{x_1}+e^{x_2}+e^{x_3}) + (1-y_1) \log \left[\frac{e^{x_2}+e^{x_3}}{e^{x_1}+e^{x_2}+e^{x_3}}\right]\)

3. Split logarithm of fraction in last term
   

   \(-\ell_1 = y_1 x_1 - y_1 \log(e^{x_1}+e^{x_2}+e^{x_3}) + (1-y_1) \log(e^{x_2}+e^{x_3}) - (1-y_1) \log(e^{x_1}+e^{x_2}+e^{x_3})\)

4. Combine like terms with \(\log(e^{x_1}+e^{x_2}+e^{x_3})\)
   

   \(-\ell_1 = y_1 x_1 + (1-y_1) \log(e^{x_2}+e^{x_3}) - \log(e^{x_1}+e^{x_2}+e^{x_3})\)

5. Take derivative with respect to \(x_1\)
   

   \(-\frac{\partial \ell_1}{\partial x_1} = y_1 - \frac{e^{x_1}}{e^{x_1}+e^{x_2}+e^{x_3}}\)

6. Recognize softmax probability definition \(p_1 = \frac{e^{x_1}}{e^{x_1}+e^{x_2}+e^{x_3}}\)
   

   \(\frac{\partial \ell_1}{\partial x_1} = p_1-y_1\)   (derivative of the log-likelihood \(\ell_1\))

This slide provides a step-by-step proof: first for the log-likelihood \(\ell\), then one sign flip for the loss \(L=-\ell\) used in training (gradient \(p-y\) w.r.t. logits).

TABLE OF CONTENTS

1. Neural Networks for Classification/Regression✓

2. The Softmax Cross Entropy Loss Function✓

3. Multi-class network: computational graph●

Neural Network As Computational Graph

Two-layer network for multi-class classification (softmax output).

• Architecture: one hidden layer (here with sigmoid activations) and one output layer whose activations are logits \(N\); applying softmax yields class probabilities \(P\) (e.g. dog / cat / horse, or 3-way softmax in the diagram).
• Task: predict a probability vector over classes — this is multi-class classification, not binary.

Neural Network As Computational Graph

This slide shows a computational graph representation of a neural network with input, hidden, and output layers.

Neural Network Computational Graph

• \(\textbf{X}\) is a 4x3 matrix (4 samples, 3 features)
• \(\textbf{W}^{(1)}\) is a 3x3 matrix
• \(\textbf{B}^{(1)}\) is a 1x3 vector
• \(\textbf{W}^{(2)}\) is a 3x3 matrix
• \(\textbf{B}^{(2)}\) is a 1x3 vector

This slide shows the detailed computational graph with matrix dimensions and operations.

Computational Graph: \(\partial L/\partial W^{(2)}_{3\times 3}\)

Compute the gradient of \(L\) (cross-entropy loss) with respect to \(W^{(2)}_{3,3}\) (backward path: \(L \leftarrow N \leftarrow W^{(2)}\); \(N\) are logits before softmax).

• \(\frac{\partial L}{\partial W^{(2)}_{3,3}} = \frac{\partial L}{\partial R_{4,3}} \odot \frac{\partial R_{4,3}}{\partial N_{4,3}} \odot \frac{\partial N_{4,3}}{\partial W^{(2)}_{3,3}}\)   (softmax + CE upstream gradient \(\partial L/\partial R = P-Y\))
• \(\frac{\partial L}{\partial R_{4,3}} = P_{4,3} - Y_{4,3}\)
• \(\frac{\partial R_{4,3}}{\partial N_{4,3}} = {\textbf{1}}_{4,3}\)
• \(\frac{\partial N_{4,3}}{\partial W^{(2)}_{3,3}} = \text{transpose}(M)_{3,4} \cdot {\textbf{1}}_{4,3}\)

Gradient for the second-layer weights: backprop stops at \(N = M W^{(2)}\) (plus bias in the full graph); local factor matches the pattern used for \(\partial U/\partial W^{(1)}\) with \(M\) in place of \(X\).

Computational Graph: \(\partial L/\partial W^{(1)}_{3\times 3}\)

Compute the gradient of \(L\) with respect to \(W^{(1)}_{3,3}\) (full backward path through hidden sigmoid: \(L \leftarrow N \leftarrow M \leftarrow V \leftarrow U \leftarrow W^{(1)}\)).

• \(\frac{\partial L}{\partial W^{(1)}_{3,3}} = \frac{\partial L}{\partial R_{4,3}} \odot \frac{\partial R_{4,3}}{\partial N_{4,3}} \odot \frac{\partial N_{4,3}}{\partial M_{4,3}} \odot \frac{\partial M_{4,3}}{\partial V_{4,3}} \odot \frac{\partial V_{4,3}}{\partial U_{4,3}} \odot \frac{\partial U_{4,3}}{\partial W^{(1)}_{3,3}}\)
• \(\frac{\partial L}{\partial R_{4,3}} = P_{4,3} - Y_{4,3}\)
• \(\frac{\partial R_{4,3}}{\partial N_{4,3}} = {\textbf{1}}_{4,3}\)
• \(\frac{\partial N_{4,3}}{\partial M_{4,3}} = {\textbf{1}}_{4,3} \cdot (W^{(2)}_{3,3})^\top\)
• \(\frac{\partial M_{4,3}}{\partial V_{4,3}} = sigmoid(V_{4,3}) \odot ({\textbf{1}}_{4,3} - sigmoid(V_{4,3}))\)
• \(\frac{\partial V_{4,3}}{\partial U_{4,3}} = {\textbf{1}}_{4,3}\)
• \(\frac{\partial U_{4,3}}{\partial W^{(1)}_{3,3}} = \text{transpose}(X)_{3,4} \cdot \textbf{1}_{4,3}\)

Chain rule for first-layer weights through the full graph from logits \(R\) back to \(U = X W^{(1)}\) (plus bias in the full graph).

Computational Graph: \(\partial L/\partial B^{(2)}_{1\times 3}\)

Compute the gradient of \(L\) with respect to \(B^{(2)}_{1,3}\) (bias added into logits \(R\); broadcast \(4\times 3\)).

• \(\frac{\partial L}{\partial B^{(2)}_{1,3}} = \frac{\partial L}{\partial R_{4,3}} \odot \frac{\partial R_{4,3}}{\partial B^{(2)}_{1,3}}\)
• \(\frac{\partial L}{\partial R_{4,3}} = P_{4,3} - Y_{4,3}\)
• \(\frac{\partial R_{4,3}}{\partial B^{(2)}_{1,3}} = {\textbf{1}}_{4,3}\)

Note. \(\odot\) gives \(4\times 3\) (one row per sample). \(B^{(2)}_{1,3}\) is shared across rows — sum across rows (column-wise over the batch) for \(\partial L/\partial B^{(2)}_{1,3}\).

Output-layer bias: same upstream factors as for \(W^{(2)}\) through \(N\); broadcast Jacobian is summarized as \({\textbf{1}}_{4,3}\) in this block notation. After multiplying the local Jacobian factors with \(\odot\), you have a \(4\times 3\) object; for a bias \(1\times 3\) you must sum across rows over the batch dimension to get \(\partial L/\partial B^{(2)}\).

Computational Graph: \(\partial L/\partial B^{(1)}_{1\times 3}\)

Compute the gradient of \(L\) with respect to \(B^{(1)}_{1,3}\) (bias in \(V\); broadcast \(4\times 3\)).

• \(\frac{\partial L}{\partial B^{(1)}_{1,3}} = \frac{\partial L}{\partial R_{4,3}} \odot \frac{\partial R_{4,3}}{\partial N_{4,3}} \odot \frac{\partial N_{4,3}}{\partial M_{4,3}} \odot \frac{\partial M_{4,3}}{\partial V_{4,3}} \odot \frac{\partial V_{4,3}}{\partial B^{(1)}_{1,3}}\)
• \(\frac{\partial L}{\partial R_{4,3}} = P_{4,3} - Y_{4,3}\)
• \(\frac{\partial R_{4,3}}{\partial N_{4,3}} = {\textbf{1}}_{4,3}\)
• \(\frac{\partial N_{4,3}}{\partial M_{4,3}} = {\textbf{1}}_{4,3} \cdot (W^{(2)}_{3,3})^\top\)
• \(\frac{\partial M_{4,3}}{\partial V_{4,3}} = sigmoid(V_{4,3}) \odot ({\textbf{1}}_{4,3} - sigmoid(V_{4,3}))\)
• \(\frac{\partial V_{4,3}}{\partial B^{(1)}_{1,3}} = {\textbf{1}}_{4,3}\)

Note. \(\odot\) gives \(4\times 3\) (one row per sample). \(B^{(1)}_{1,3}\) is shared across rows — sum across rows (column-wise over the batch) for \(\partial L/\partial B^{(1)}_{1,3}\).

Hidden-layer bias: same chain as for \(W^{(1)}\) through \(U\), with the last local factor \({\textbf{1}}_{4,3}\) for the broadcast add of \(B^{(1)}\) into \(U\). After the \(\odot\) chain you still have shape \(4\times 3\); sum across rows over the batch to reduce to \(\partial L/\partial B^{(1)}_{1,3}\) (\(1\times 3\)).

Thank You!